Papers with learning efficiency
RTSM: Knowledge Distillation with Diverse Signals for Efficient Real-Time Semantic Matching in E-Commerce (2025.naacl-industry)
Copied to clipboard
| Challenge: | e-commerce product search is a key component of product discovery and sales in ecommerce . high computational demands of large transformer models pose challenges for their deployment in real-time scenarios. |
| Approach: | They propose a framework for real-time semantic matching that leverages both soft labels from a teacher model and ground truth generated from pairwise query-product and query-query signals. |
| Outcome: | The proposed framework outperforms teacher models and state-of-the-art models on e-commerce datasets. |
Pre-training Language Model as a Multi-perspective Course Learner (2023.findings-acl)
Copied to clipboard
Beiduo Chen, Shaohan Huang, Zihan Zhang, Wu Guo, Zhenhua Ling, Haizhen Huang, Furu Wei, Weiwei Deng, Qi Zhang
| Challenge: | Experimental results show that our method significantly improves ELECTRA’s average performance by 2.8% and 3.2% absolute points respectively on GLUE and SQuAD 2.0 benchmarks. |
| Approach: | They propose a multi-perspective course learning method to fetch many degrees and visual angles for sample-efficient pre-training and to fully leverage the relationship between generator and discriminator. |
| Outcome: | The proposed method improves ELECTRA's performance on GLUE and SQuAD 2.0 benchmarks and overshadows recent advanced ELECL-style models under the same settings. |
Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding (2025.acl-srw)
Copied to clipboard
| Challenge: | Existing curriculum learning approaches rely on manually defined difficulty metrics which may not accurately reflect the model’s own perspective. |
| Approach: | They propose a self-adaptive curriculum learning paradigm that prioritizes fine-tuning examples based on difficulty scores predicted by pre-trained language models (PLMs) they evaluate four datasets covering binary and multi-class classification tasks. |
| Outcome: | The proposed model leads to faster convergence and improved performance compared to standard random sampling. |
Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Neural language models (LMs) are trained on orders of magnitude more language data than human language learners receive, but without supervision from other sensory modalities that play a crucial role in human learning. |
| Approach: | They propose a grounded language learning procedure that leverages visual supervision to improve textual representations. |
| Outcome: | The proposed procedure outperforms standard language-only models in terms of learning efficiency in small and developmentally plausible data regimes and improves perplexity by around 5% on multiple language modeling tasks compared to other models trained on the same amount of text data. |
A Natural Bias for Language Generation Models (2023.acl-short)
Copied to clipboard
| Challenge: | a standard probabilistic model for language generation has likely not yet learnt many semantic or syntactic rules of natural language, making it difficult to estimate the probability distribution over next tokens. |
| Approach: | They propose to initialise bias terms in a model's final linear layer with the log-unigram distribution and use it to output the unigram frequency statistics as prior knowledge. |
| Outcome: | The proposed method improves learning efficiency and improves overall performance. |
Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning (2026.eacl-long)
Copied to clipboard
| Challenge: | Recent advances in reinforcement learning with verifiable rewards (RLVR) show that large language models enhance their reasoning abilities when trained with veriable signals. |
| Approach: | They propose a method for a problem-aware filtering system that maximizes learning efficiency by selecting tasks of intermediate difficulty. |
| Outcome: | The proposed model improves when trained with verifiable rewards, but training efficiency is bottleneck . the proposed model achieves +12% gains in less than half the training steps of standard GRPO . |
Dynamic Knowledge Distillation for Pre-trained Language Models (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods conduct knowledge distillation statically, e.g., student model aligns output distribution to teacher model on pre-defined training dataset. |
| Approach: | They propose a dynamic knowledge distillation that empowers the student to adjust the learning procedure according to its competency . they find it is promising and provide discussions on potential future directions towards more efficient methods . |
| Outcome: | The proposed method can boost student model performance while accelerating training . the proposed method reduces memory usage and accelerates model inference . |
Meta-Reasoning: Semantics-Symbol Deconstruction for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods rely on syntactically mapping natural languages to complete formal languages like Python and SQL. |
| Approach: | They propose to deconstruct reasoning-independent semantic information into generic symbolic representations, thereby efficiently capturing more generalized reasoning knowledge. |
| Outcome: | The proposed method improves in-context reasoning accuracy, learning efficiency, out-of-domain generalization, and output stability compared to the Chain-of thought technique. |
Personality-aware Student Simulation for Conversational Intelligent Tutoring Systems (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) can be adopted as tutoring agents for math and language learning. |
| Approach: | They propose a framework to construct profiles of different student groups by refining and integrating both cognitive and noncognitive aspects, and leverage LLMs for personality-aware student simulation in a language learning scenario. |
| Outcome: | The proposed framework can construct profiles of different student groups by refining and integrating both cognitive and noncognitive aspects, and leverage LLMs for personality-aware student simulation in a language learning scenario. |
A Versatile Adaptive Curriculum Learning Framework for Task-oriented Dialogue Policy Learning (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing training paradigms for dialogue policy learning with brute-force random sampling are expensive and lack reliable evaluation of difficulty scores. |
| Approach: | They propose a flexible adaptive curriculum learning framework that integrates curriculum learning with a generic global curriculum. |
| Outcome: | The proposed framework improves learning performance and efficiency on three public dialogue datasets. |
TELLME: Test-Enhanced Learning for Language Model Enrichment (2026.findings-eacl)
Copied to clipboard
Minjun Kim, Inho Won, HyeonSeok Lim, MinKyu Kim, Junghun Yuk, Wooyoung Go, Jongyoul Park, Jungyeul Park, KyungTae Lim
| Challenge: | Continual pre-training (CPT) has been widely adopted as a method for domain expansion in large language models, but has faced challenges such as acquiring large-scale domain-specific datasets and high computational costs. |
| Approach: | They propose a method that integrates the Test-Enhanced Learning principle with CPT to promote efficient domain-specific knowledge acquisition and long-term memory retention. |
| Outcome: | The proposed method outperforms existing methods by 23.6% in the financial domain and achieves 9.8% improvement in long-term memory retention. |
SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Text-based methods lag behind graph embedding-based approaches for knowledge graph completion (KGC) |
| Approach: | They propose three types of negatives to improve contrastive learning to improve learning efficiency. |
| Outcome: | The proposed model outperforms embedding-based methods on several benchmark datasets. |
Visual-Language Navigation Pretraining via Prompt-based Environmental Self-exploration (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods of fine-tuning vision-language navigation models require extra human-labeled data and lack self-exploration capabilities in environments. |
| Approach: | They propose a method that can self-explore environments without human labeling . they use a large-scale cross-modal pretrained model to build an in-domain dataset . |
| Outcome: | The proposed model can self-explore environments without human labeling without human supervision and generates structured instructions without human intervention. |
InvestAlign: Overcoming Data Scarcity in Aligning Large Language Models with Investor Decision-Making Processes Under Herd Behavior (2025.acl-long)
Copied to clipboard
| Challenge: | relying on authentic data for Supervised Fine-Tuning (SFT) is costly and expensive. |
| Approach: | They propose a framework that constructs high-quality SFT datasets by leveraging theoretical solutions to similar and simple optimal investment problems rather than the complex scenarios. |
| Outcome: | The proposed framework achieves faster parameter convergence than using real-user data, suggesting superior learning efficiency. |
Adaptive and Personalized Exercise Generation for Online Language Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Empirical studies have shown various benefits of adaptive learning, such as improved student learning outcomes, lower dropout rates, and increased instructor satisfaction. |
| Approach: | They propose to combine a knowledge tracing model that estimates each student’s evolving knowledge states from their learning history with a controlled text generation model that generates exercise sentences based on the student’ s current estimated knowledge state and instructor requirements of desired properties. |
| Outcome: | The proposed model can generate superior exercises based on student state and instructor requirements . Empirical studies have shown that adaptive learning improves student learning outcomes, lower dropout rates, and increased instructor satisfaction. |
GLTW: Joint Improved Graph Transformer and LLM via Three-Word Language for Knowledge Graph Completion (2025.findings-acl)
Copied to clipboard
Kangyang Luo, Yuzhuo Bai, Cheng Gao, Shuzheng Si, Zhu Liu, Yingli Shen, Zhitong Wang, Cunliang Kong, Wenhao Li, Yufei Huang, Ye Tian, Xuantang Xiong, Lei Han, Maosong Sun
| Challenge: | Existing knowledge graphs lack the ability to integrate structural information into LLMs and output predictions deterministically. |
| Approach: | They propose a method which encodes structural information of KGs and merges it with LLMs to enhance KGC performance. |
| Outcome: | The proposed method improves the performance of KG Completion datasets on KGs by integrating structural information with LLMs. |
R-PRM: Reasoning-Driven Process Reward Modeling (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing Process Reward Models (PRMs) output evaluation scores directly, limiting both learning efficiency and evaluation accuracy. |
| Approach: | They propose a Reasoning-Driven Process Reward Modeling (R-PRM) which activates inherent reasoning to enhance process-level evaluation. |
| Outcome: | The proposed model outperforms baseline models on ProcessBench and PRMBench by 13.9 and 8.5 F1 scores. |
KARL: Knowledge-Aware Retrieval and Representations aid Retention and Learning in Students (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing student models use study data like student's past responses to predict the probability a student can recall a flashcard. |
| Approach: | They propose to use student models to predict recall of flashcards to build a content-aware student model that uses deep knowledge tracing, retrieval, and BERT to predict student recall. |
| Outcome: | The proposed content-aware student model outperforms existing student models in AUC and calibration error and is more efficient than SOTA. |
EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning for LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to reinforcement learning with verifiable reward (RLVR) are limited by difficulty or lack of exploration. |
| Approach: | They propose a self-evolving curriculum learning framework based on chain-of-thought reasoning optimization that constrains exploration space by self-generating and verifying CoT trajectories. |
| Outcome: | The proposed framework enables LLMs to solve previously unsolved problems without external supervision and is compatible with various RL fine-tuning methods. |
Video-MMMU: Evaluating Knowledge Acquisition from Multidisciplinary Professional Videos (2026.acl-long)
Copied to clipboard
| Challenge: | Existing video benchmarks do not evaluate the knowledge acquisition capabilities of Large Multimodal Models (LMMs) existing video benchmark focuses on static, general visual understanding tasks, without evaluating whether models can acquire knowledge dynamically. |
| Approach: | They propose a multi-modal, multi-discipline, multitrack benchmark that evaluates Large Multimodal Models’ ability to acquire knowledge from college-level, educational videos. |
| Outcome: | The proposed benchmark reveals a substantial gap between human learners and current Large Multimodal Models (LMMs) and focuses on improving their learning efficiency. |